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INTRODUCTION 

One mission of the Paul M. Fitts Human Engineering Division is to improve the user interface 
for complex systems through user-centered exploratory development and research activities. In 
support of this goal, many current projects attempt to advance and exploit user-interface concepts made 
possible by so-called Virtual Reality (VR) technologies. Virtual environments may be used as a general 
purpose interface medium, an alternative display/control method, a data visualization and analysis tool, 
or a graphically based performance assessment tool. All of these uses of VR may be exploited in the 
development of new user-interface prototypes and supporting design tools. As a result, the Division 
has several active R&D efforts in these areas, as they pertain to user-machine interfaces. 

The purpose of this presentation is to provide a brief overview of the range of R&D projects 
within the Division that involve VR technology. For the purpose of discussion, research projects are 
clustered into four categories: 

• Prototype Interface Hardware/Software Development 

• Integrated Interface Concept Development 

• Interface Design and Evaluation Tool Development 

• User and Mission Performance Evaluation Tool Development. 
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PROTOTYPE INTERFACE HARDWARE SOFTWARE DEVELOPMENT: 

HELMET-MOUNTED DISPLAY SYSTEMS 


The Division has been a pioneer in the development of visually coupled systems (VCS) 
technology which centers around the design of helmet-mounted display (HMD) devices. Beginning in 
1966, a line of HMDs has been produced, ranging from small field-of-view (FOV) monocular systems 
to wide FOV stereoscopic ones. R&D activities address the development of high luminance miniature 
CRTs, relay optical systems, head gear, militarized image generators, trackers, and all technological 
and engineering aspects required to produce fully integrated systems for use in airborne platforms. The 
technical advancements made in this work over die years have provided the foundation from which the 
virtual reality industry has emerged. For example, the need for a method to reliably track head position 
without restricting operator movement or emergency egress sdmulaied the development of the magnetic 
tracking concept now used in many VR systems. TTie first successful implementation of such a device 
was sponsored by the Division and achieved in collaboration with our staff. 

Current VCS development efforts are concentrated on: 

• standardized, plug compatible VCS systems for integration into military platforms 

• color CRT that can operate in high (luminance) ambient environments 

• system and component performance measurement methods and standards 

• psychophysical performance assessment . 





Figure 1 
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VCS CONTROL/DISPLAY CONCEPTS 


In addition to developing the hardware and software technology infrastructure for the VCS 
system, we also develop advanced interface concepts around the technology. One example is the 
design of an interface device that supports head aiming (to select and track targets) and head-referenced 
instrument displays (flight and weapons) that are always available to the operator. This work involves 
the design of display formats, symbology, and control methods. It also includes a wide range of 
human performance research to insure compatibility of the concept to both the user and the expected 
real world task conditions. Several concepts have been developed and evaluated in simulated and 
actual flight tests. 



Figure 2 
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THREE-DIMENSIONAL AUDIO DISPLAYS 


Similar to the VCS R&D efforts, Armstrong Laboratory has also been a leader in the 
development of 3-D audio systems, including hardware concepts, psychophysical analysis, application 
systems, and flight testing. Our first localized display was designed and fabricated in-house in the late 
1980's. New systems compatible with operational avionics systems are being developed and 
evaluated. 

The geodesic dome shown below is housed in an anechoic chamber located in the laboratory. 
This is the only available facility in the country that can support the full range of technical studies 
needed to advance this technology. 

A study was recently completed that addressed target acquisition performance with and without 
the aid of a 3-D sound localizer. A summary of the results of that study is shown in the next two 
charts. Each graph depicts target search times as color zones (shown here as shapes of gray) over the 
search space defined in terms of azimuth and elevation angle. The first graph displays times when 
search was completed visually without the aid of localized sound cueing. The next graph shows the 
search time map when localize sound cueing was used. The improvement with sound cueing is 
obvious from visual inspection of the graphs. 
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Figure 3 
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Figure 5 
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ALTERNATIVE CONTROL TECHNOLOGIES 


Several projects in the Division are involved with developing and exploring the feasibility of 
novel interface methods as alternative I/O devices for use in a wide variety of application domains. 
These include the development of a gesture recognition system that combines a data glove which senses 
figure postures and gestures (coupled with a position/orientation tracker) with a neural net classifier. A 
demonstration system is being constructed that recognizes American sign language. The device may be 
used as an input mechanism to a system/computer or as a means to support nonverbal communication 
between people. The Veterans Administration has expressed interest in this work as an assistive 
technology. 

A facial sensor testbed has been developed to determine if continuous speech recognition 
accuracy can be improved by “reading” facial expressions during speech. The basic idea is to use facial 
patterns as a basis for disambiguating classification uncertainties found with current hidden Markov 
model speech recognition systems. 

Perhaps the most unusual alternative control technology under study is called Brain Actuated 
Control (BAC). As the name suggests, we are exploring ways brain states can be detected and used as 
a control decision which is then executed by the use of standard servomechanism principles in a closed- 
loop system. We have demonstrated reliable two-state based brain control, with the operator 
voluntarily enhancing and suppressing the magnitude of a certain EEG frequency. Continuous control 
has also been demonstrated through exploitation of a composite voluntarily produced EEG-EMG 
signal. This is another area where our work has spin-off potential for use with disadvantaged 
individuals. 


Facial Sensor Test Bed Hand Gesture Recognition 



Figure 6 


130 



Figure 7 


Brain Actuated Flight Simulator 
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INTEGRATED INTERFACE CONCEPT DEVELOPMENT 


Beginning with the Visually Coupled Airborne Simulation System (VCASS), which became 
operational in the early 1980’s, we have had the ability to produce virtual interfaces or virtual cockpit 
concepts for investigation. This system probably marked the beginning of the new era of VCS 
technology that has become known as Virtual Reality, Virtual Environment, or Synthetic Environment 
technology. VCASS provides a 120 degree (horizontal) by 60 degree (vertical) instantaneous FOV in a 
head-mounted stereoscopic display system. We have coupled it over the years with data gloves, a 
simple force reflecting device, and 3-D audio to produce various forms of virtual cockpits. 
Unfortunately, limitations in computer graphics and system time delays have interfered with our ability 
to adequately assess the value of these innovative interfaces. 

We have recently opened a new facility called the Synthesized Immersion Research 
Environment (SIRE), that provides a full capability immersive VR system. It integrates haptic displays 
with 3-D sound and VCS technology and also includes a large screen projection system. SIRE 
supports a wide range of human performance research, including investigations of augmented cockpit 
concepts which couple VR concepts and alternative control concepts with more conventional panel 
mounted displays. 
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INTEGRATED INTERFACE CONCEPT DEVELOPMENT 



Figure 8 
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CONTROL INTEGRATION 


INTEGRATED INTERFACE CONCEPT DEVELOPMENT 



Figure 9 


134 


HAND 




Cognitive 

Assessment 


INTEGRATED INTERFACE CONCEPT DEVELOPMENT 



Figure 10 
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INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT 


Historically, it has often proven difficult to transition user-centered interface ideas and concepts 
into operational systems. There are many reasons for this breakdown in the development process. 
Most agree that one reason is the lack of adequate interface design and evaluation tools that fit properly 
into the systems engineering process used in advanced design of large-scale systems. In an effort to 
overcome this difficulty, we have been developing tools and concepts to support the design and 
evaluation process. 

One area of research important to this goal is engineering anthropometry. In this area, we 
perform studies that allow us to develop relevant anthopometric data bases. For example, we assess 
the ability of different body types (size, strength, etc.) to perform different actions often required in 
maintenance tasks. Data bases like these have been combined with computer graphics systems to 
produce interactive visualization and analysis tools for assessing system designs (computer 
representations). VR has recently been used to allow the designer to assume different body sizes 
during the analysis. 
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INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT 



Figure 1 1 










INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT 



Figure 12 
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INTERFACE DESIGN AND EVALUATION TOOL DEVELOPMENT 



Figure 13 
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HUMAN CENTERED DESIGN ON THE CAD/CAE PIPELINE 


Another effort has concentrated on the development of active information using multi-media 
technology. The goal is to be able to place human performance data into the CAD/CAE environment 
used in design. This research has addressed ways to improve access to information, ways to help the 
designer understand what questions they need to ask, and ways to facilitate opportunistic searches. In 
addition, we have invested in the production of a new tool called a perception and performance 
prototyper. This software product allows the designer to transition from equations and graphical 
presentations and analysis to experiential methods that allow the designer to experience the perceptual 
and performance consequences of their design decisions. 



Figure 14 
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Figure 15 



HUMAN CENTERED DESIGN ON THE CAD/CAE PIPELINE 



Figure 16 
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USER PERFORMANCE AND VIRTUAL SYSTEMS 


It is clear that the potential applications for Virtual Technology are enormous. DoD, for 
example, is moving ahead with the use of this technology as both a tool and an end product in the 
development of advanced systems. Distributive Interactive Simulation (DIS) is being considered as a 
test bed to investigate advanced concepts under operational like conditions. Some of these systems 
may be virtual; others physical. In this and all other work involving VR technology, we suffer today 
from a lack of adequate understanding of how user performance interacts with properties of the VR 
technology itself. Without information like this we will not be able to determine to what degree user 
performance in, say, a DIS environment, reflects what can be expected in the real world, and what is 
due to characteristics of the VR simulations. Information of this nature is also needed to aid in design 
decision making, and to provide performance-based bench marks or requirements to guide VR 
technologists in advancing the state of the art. 

We have undertaken a research project that focuses on this issue. One goal of this effort is to 
establish a quantitative relation between VR system properties and user performance. Our plan is to do 
this with a series of standardized bench mark tasks that cover a wide range of perception, perceptual- 
motor, and cognitive activities. To date we have completed studies that demonstrate the cost of VR 
system time delay on manual tracking and control and aimed movement control. The effects of several 
VR system variables on size-distance judgments in a virtual environment have also been studied. 

The next chart depicts the fact that user performance based design data requires a link to be 
made between properties of user performance with properties of VR systems. The user performance 
properties must, in turn, be linked to specific application domains in order to map user performance to 
task performance. One goal of our research is to provide the desired user-performance data and the 
tools needed to complete the mapping to a wide variety of task domains. 

Figures 18 and 19 provide a brief summary of the results from some experiments that 
investigated simple aimed movement performance which was accomplished in a virtual environment 
(VE). In general, the data follow Fitts Law over an ID range of 2 to 6, but show departure from 
linearity around an ID of 7. In comparison with physical aimed movement, performance by highly 
practiced subjects was degraded by about 35-50% in a VE (see Fig. 19). Further, the effect of time 
delay in the VR system caused movement times to increase substantially under ID 5 conditions, 
somewhat for ID 6 conditions, and very little at other ID levels (see Fig. 19). Data like these provide 
the basis for constructing design trade-off nomographs like the example shown in Fig. 20. 

A design trade-off nomograph (see Fig. 20) is a form of data representation that highlights the 
relation between a measure of user performance and two or more measures of VR system properties. 
This form of representation allows a VR system designer to easily see how user performance interacts 
with system properties. It can be used for defining system requirements and for performing trade-off 
analyses to select between alternative design options that can meet a requirement. Thus, it aids the 
designer in seeing the connection between user performance in a way that is clearly connected to 
system properties, but at the same time allows the designer to bring in technology risk, cost, and 
schedule factors in the process of deciding how to meet a performance goal. 
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USER PERFORMANCE AND VIRTUAL SYSTEMS 



Figure 17 
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Figure 20 
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CLOSING REMARKS 


As you can see, Armstrong Laboratory is engaged in a broad range of R&D research that 
involves both the development and exploitation of VR technology. We have a long history with the 
technology which has allowed us to achieve a high degree of understanding of the technical issues and 
challenges that need to be solved to insure its effective use in different application domains. We are 
very sensitive to the fact that VR technology by itself is neither desirable or undesirable. Rather, its 
value depends on how well we can unite it with the human user to produce successful products. 



DETERMINING REQUIREMENTS AND 
CAPABILITIES OF VIRTUAL INTERFACES 

Figure 21 
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